System for increasing utilization of storage media

ABSTRACT

A storage system creates an abstraction of flash Solid State Device (SSD) media allowing random write operations of arbitrary size by a user while performing large sequential write operations of a uniform size to an SSD array. This reduces the number of random write operations performed in the SSD array and as a result increases performance of the SSD array. A control element determines when blocks from different buffers should be combined together or discarded based on fragmentation and read activity. This optimization scheme increases memory capacity and improves memory utilization and performance.

RELATED APPLICATIONS

This application is a continuation application of, and claims priorityto, application Ser. No. 12/759,644, entitled: SYSTEM FOR INCREASINGUTILIZATION OF STORAGE MEDIA, filed on Apr. 13, 2010, which claimspriority to provisional application Ser. No. 61/170,472, entitled:STORAGE SYSTEM FOR INCREASING PERFORMANCE OF STORAGE MEDIA, filed Apr.17, 2009 each of which is incorporated by reference in its entirety.

BACKGROUND

Storage systems typically present a plurality of physical media devicesas one or more logical devices with desirable advantages over theoriginal physical media. These advantages can be in the form ofmanageability (performing per device operations to a group of devices),redundancy (allowing and correcting media errors on one or more devicestransparently), scalability (allowing the size of logical devices tochange dynamically by adding more physical devices) or performance(using parallelism to spread storage operations over multiple mediadevices). Additionally, storage systems may employ intelligentoperations such as caching, prefetch or other performance-enhancingtechniques.

For comparative purposes, storage systems are described in terms ofcapacity and performance. Capacity is described in terms of bytes (basicunit of computer storage—conceptually equivalent to one letter on atyped page) or blocks where a block is typically 512 Bytes. The numberof bytes in a storage system can be very large (several million millionsof bytes—or terabytes). Performance of a storage device is typicallydependent of the physical capabilities of the storage medium. Thisperformance is typically considered in terms of three parameters:Input/Output Operations per Second (IOPs), throughput (bytes per secondthat can be accessed) and latency (time required to perform a nominalaccess). The IOPs metric is further described for both sequential andrandom access patterns.

Configuration of a storage system allows for selective optimization ofcapacity and performance. Capacity optimization is achieved by simplyaggregating the capacity of all physical devices into a single logicaldevice. This logical device will have higher capacity than theconstituent devices but equivalent or slightly lower performance.Reliability optimization may involve using replication that sacrificeshalf the capacity. Alternatively, reliability optimization may involvesome error correction encoding which sacrifices some capacity but lessthan that from replication. Performance optimization may involveduplication which allows twice as many read operations per unit timeassuming some balancing mechanism, striping which increases throughputby spreading operations over an array of devices, or caching which usesmemory to act as a buffer to the physical media. In general, the storagesystem will optimize for a desired performance metric at the cost ofanother or by incorporating additional physical elements (such as logic,memory or redundancy) beyond the component devices.

Determining the optimal, or most suitable, configuration of a storagesystem requires matching the demands of the user of the system to thecapabilities of the physical devices and the optimization capabilitiesof the storage system. The performance of the constituent physicaldevices is typically the determining factor. As an example, commonstorage systems typically favor IOPs over capacity and thus choose touse a large number of smaller capacity disks vs. creating the equivalentaggregate capacity from larger capacity devices. As media technologyevolves, new methods of increasing performance and compensating forshortcomings of the physical media are constantly sought.

A physical media may take the form of Solid State Storage technologyknown as Multi-Level Cell (MLC) NAND flash. The MLC NAND flash iscommonly used in cameras, portable devices such as Universal Serial Bus(USB) memory sticks, and music players as well as consumer electronicssuch as cellular telephones. Other forms of flash in common use includeSingle-Level Cell (SLC) NAND flash and NOR flash. Both of these lattertypes offer higher performance at a significantly higher cost ascompared to MLC NAND flash. Many manufacturers are currently offeringNAND flash with an interface that mimics that of traditional rotatingstorage devices (disk drives). These flash devices are referred to asflash Solid State Drives (SSDs) and may be constructed using either MLCor SLC technology.

Flash SSD devices differ from traditional rotating disk drives in anumber of aspects. Flash SSD devices have certain undesirable aspects.In particular, flash SSD devices suffer from poor random writeperformance that degrades over time. Because flash media has a limitednumber of writes (a physical limitation of the storage material thateventually causes the device to “wear out”), write performance is alsounpredictable.

Internally, the flash SSD will periodically rebalance the writtensections of the media in a process called “wear leveling”. This processassures that the storage material is used evenly thus extending theviable life of the device. The inability to anticipate, or definitivelyknow, when and for how long such background operations may occur (lackof transparency) is a principal cause of the performance uncertainty.

For example, a user cannot typically access data in the flash SSD devicewhile these rebalancing operations are being performed. The flash SSDdevice does not provide prior notification of when the backgroundoperations are going to occur. This prevents an application fromanticipating the storage non-availability and scheduling other tasksduring the flash SSD rebalancing operations. However, the significantperformance advantage of flash SSDs over rotating media in random andsequential read operations makes SSDs ideal media for high performancestorage systems, if the write performance issues can be overcome oravoided.

It has also been determined that although the random write performanceof the SSDs for a common write operation size of 4 KB (4 thousand bytesor 8 blocks) was poor, the sequential write performance for large writeoperations above 1 MegaBytes (1 million bytes) was acceptable providedthat all writes were of the same size. When always servicing writes ofuniform size, the SSD can minimize the amount of background activity.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a storage system used for accessing a SolidState Device (SSD) array.

FIG. 2 shows in more detail some of the operations performed by thestorage system shown in FIG. 1.

FIG. 3 is a flow diagram showing in more detail how the storage systemoperates.

FIG. 4 is a block diagram showing a control element used in the storagesystem of FIG. 1.

FIG. 5 is a block diagram showing an example write operation performedby the storage system.

FIG. 6 shows how the control element tracks data utilization.

FIG. 7 is a flow diagram showing in more detail the operations performedby the control element during a write operation.

FIG. 8 is a flow diagram showing in more detail the operations performedby the control element during a read operation.

FIG. 9 is a flow diagram showing in more detail the operations performedby the control element during a data invalidate operation.

FIG. 10 is a block diagram showing how the control element combinestogether data from different buffers.

FIG. 11 is a flow diagram showing in more detail the operationsperformed by the control element in FIG. 10.

FIG. 12 is a flow diagram showing how the control element ranksutilization of buffers.

DETAILED DESCRIPTION

A novel storage system includes an indirection mechanism and controlelement. The storage system creates an abstraction of flash Solid StateDevice (SSD) media allowing random write operations of arbitrary size bya user while performing large sequential write operations of a uniformsize to an SSD array. This reduces the number of random write operationsperformed in the SSD device and as a result reduces performancedegradation in the SSD device. The uniform block writes to the SSDdevice can also increase storage throughput since the SSD device has toperform fewer defragmentation operations. A defragmentation operation isa type of background activity that can involve a number of internal readand write operations blocking normal user access to the SSD.

The storage system increases storage availability by using transparencyand a handshaking scheme that allows users to eliminate or minimize thebackground operations performed in an SSD array. The storage system alsoprovides the user with the actual physical addresses where data isstored in the SSD array via the indirection mechanism. This is differentthan conventional SSD arrays where data indirection and the physicaladdresses for stored data are hidden from the user. Read operations aremonitored for each of the different SSD devices in the SSD array. Afirst SSD device may be read more often than a second SSD device. Thestorage system may write new data blocks into the second SSD device,even when the second SSD device is currently storing more data than thefirst SSD device. This can increase throughput in the SSD array forparticular applications where data is typically read from memory moreoften than written to memory.

For example, a web server may provide web pages to clients. New webpages may infrequently be written into memory by the web server.However, the same web server may constantly read other web pages frommemory and supply the web pages to clients. Thus, writes to differentSSD devices may be performed based on the type of SSD deviceutilization, not solely on SSD device capacity. An optimal performancebalance is reached when all SSD devices experience the same read demand.It is possible, and very likely, that different write loads would berequired to achieve this balance.

The storage system can be configured to use different block sizes forwriting data into the SSD array according to performance characteristicsof the SSD devices. For example, a particular SSD device may be able toperform a single 4 Mega Byte (MB) write significantly faster than 10004K block writes. In this situation, the storage system might beconfigured to perform all writes to the SSD array in 4 MB blocks, thusincreasing the total available write throughput of the SSD array. All 4Kblock writes would have to be pieced together (aggregated) into a single4 MB write to achieve this increase.

In another embodiment, a control element determines when blocks fromdifferent buffers should be combined together or discarded based onfragmentation and read activity. This optimization scheme increasesmemory capacity and improves memory utilization. Optimizing thecombination requires aggregating smaller writes into larger writeswithout wasting available space within the larger write. Maintaining theinformation of all smaller writes is the function of the controlelement.

FIG. 1 shows a storage system 100 that includes an indirection mechanism200 and a control element 300. The storage system 100 uses the SSDoperating characteristics described above to improve storageperformance. In one embodiment, the storage system 100 and storage users500 are software executed by one or more processors 105 and memorylocated in a server 502. In other embodiments, some elements in thestorage system 100 may be implemented in hardware and other elements maybe implemented in software.

In one embodiment, the storage system 100 is located between the users500 and a disk 20. The storage system 100 can be a stand-aloneappliance, device, or blade, and the disk 20 can be a stand-alone diskstorage array. In this embodiment, the users 500, storage system 100,and disk 20 are each coupled to each other via wired or wirelessInternet connections. In another embodiment, the users 500 may accessone or more disks 20 over an internal or external data bus. The storagesystem 100 in this embodiment could be located in the personal computeror server, or could also be a stand-alone device coupled to thecomputer/client via a computer bus or packet switched networkconnection.

The storage system 100 accepts reads and writes to disk 20 from users500 and uses the SSD array 400 for accelerating accesses to data. In oneembodiment, the SSD array 400 could be any combination of Dynamic RandomAccess Memory (DRAM) and/or Flash memory. Of course, the SSD array 400could be implemented with any memory device that provides relativelyfaster data access than the disk 20.

The storage users 500 include any software application or hardware thataccesses or “uses” data in the SSD array 400 or disk array 20. Forexample, the storage users 500 may comprise a cache application used byan application 504 operated on a storage server 502. In this example,application 504 may need to access data stored in SSD array 400responsive to communications with clients 506 via a Wide Area Network(WAN) 505 or Local Area Network (LAN) 505 referred to generally as theInternet.

In one embodiment, the storage users 500, storage system 100, and SSDarray 400 may all be part of the same appliance that is located in theserver or computing device 502. In another example, any combination ofthe storage users 500, storage system 100, and SSD array 400 may operatein different computing devices or servers. In other embodiments, thestorage system 100 may be operated in conjunction with a personalcomputer, portable video or audio device, or some other type of consumerproduct. Of course these are just examples, and the storage system 100can operate in any computing environment and with any application thatneeds to write and read date to and from memory devices.

The storage system 100 presents the SSD array 400 as a logical volume tostorage users 500. Storage system 100 presents logical blocks 150 ofvirtual storage that correspond to physical blocks 450 of physicalstorage in SSD array 400. The SSD array 400 consists of a plurality ofSSD devices 402, two of which are referenced as SSD device 402A and SSDdevice 402B. The total number of SSD devices 402 in SSD array 400 maychange over time. While shown being used in conjunction with an SSDarray 400, it should also be understood that the storage system 100 canbe used with any type or any combination of memory devices.

Storage users 500 may consist of a number of actual users or a singleuser presenting virtual storage to other users indirectly. For example,as described above, the storage users 500 could include a cacheapplication that presents virtual storage to a web application 504operating on the web server 502. The logical volume presented to theusers 500 has a configurable block size which is considered fixed duringthe normal operating mode.

The size of the virtual blocks 150, a block size for transfers betweenthe storage system 100 and SSD array 400, and the scheme used forselecting SSD devices 402 is contained within configuration registers110. Upon initialization, storage system 100 interprets theconfiguration data in register 110 to set configuration parameters. Forthe purpose of subsequent examples, the virtual block size 150 isassumed to be configured as 4 KB. Read and write operations performed bystorage system 100 reference an integral number of the virtual blocks150 each of size 4 KB.

The indirection mechanism 200 is operated by the storage users 500 andis populated by the control element 300 with the physical addresseswhere data is located in SSD array 400. Indirection mechanism 200consists of an indirection table 220 consisting of a plurality ofindirection entries 230, two of which are referenced as indirectionentry 230A and indirection entry 230B. In one embodiment, indirectiontable 220 consists of a block level index representation of a logicalstorage device. The index representation allows virtual blocks 150 to bemapped to physical blocks 450 in SSD array 400. This requires one entryper virtual block 150 of logical storage or the ability to uniquely mapany block of logical storage to a block of physical storage in SSD array400.

In another embodiment, indirection mechanism 200 consists of a searchstructure, such as a hash, binary tree or other structure, such that anyphysical block 450 within the SSD array 400 can be mapped to a uniqueindirection entry 230 associated with a unique virtual block 150. Thissearch structure may be constructed in situ as the storage media 400 isutilized (written). In this embodiment, indirection table 220 grows asmore unique virtual blocks 150 are written to the storage system 100.

In another embodiment, indirection table 220 consists of a multi-levelbitmap or tree search structure such that certain components are staticin size while other components grow as more unique virtual blocks 150are created in the storage system 100. In another embodiment,indirection mechanism 200 is implemented as a hardware component orsystem such as a content addressable memory (CAM). In this embodiment,multiple levels of indirection may be used, some of which are embodiedin software.

All embodiments of indirection mechanism 200 resolve a block address ofa read or write operation from users 500 into a unique indirection entry230. The indirection entry 230 consists of a SSD device ID 232, useraddress 233, block address 234, and a block state 236. The SSD device ID232 corresponds to a unique SSD device 402 in SSD array 400. Blockaddress 234 corresponds to the unique physical address of a physicalblock 450 within the SSD device 402 that corresponds with the device ID232. A block refers to a contiguous group of address locations withinthe SSD array 400. Block state 236 contains state information associatedwith block address 234 for device ID 232. This block state 236 mayinclude, but is not limited to, timestamp information, validity flags,and other information.

In one embodiment, device ID 232 and block address 234 correspond tophysical SSD devices 402 through a secondary level of indirection. Inthis embodiment, a disk controller (not shown) may be used to createlogical devices from multiple physical devices.

In subsequent description, the choice of blocks of size 4 KB and buffersof size 4 MB is used extensively. The example of a 4 KB block size and 4MB buffer size is used for explanation purposes. Both block and buffersizes are configurable and the example sizes used below are not intendedto be limiting. Chosen sizes as well as the ratio of sizes may differsignificantly without compromising the function of the presentembodiments.

Overall Operation

FIGS. 1-3 and particularly FIG. 3, in a first operation 250 the storageuser 500 writes data 502 of a random size without a specified SSDaddress to the storage system 100. Data 502 does contain a user addresswhich will used in the future to read data 502. In operation 252, thecontrol element 300 assigns the random write data 502 to one or more 4KB blocks 508 within a 4 MB staging buffer 370.

The control element 300 also identifies a SSD device 402 within that SSDarray 400 for storing the contents of 4 MB buffer 370. The controlelement 300 in operation 254 notifies the indirection mechanism 200 ofthe particular SSD device 402 and physical block address where the data502 is written into the SSD array 400. The user address 233 specified aspart of the write of data 502 is stored within indirection mechanism 200in such a way that a lookup of the user address 233 will return thecorresponding physical block address 234. Storage user 500 cansubsequently retrieve data 502 using this physical block address. Inoperation 256, the data 502 in the staging buffer 370 is written intothe SSD array 400.

Although the user has not specified an SSD address for data 502, someimplementation specific transaction state may exist. In one embodiment,the user submits multiple instances of write data 502 serially, awaitinga returned physical block address for each write and recording thisaddress within a memory. In another embodiment, the user submits severalinstances of write data 502 concurrently along with a transactiondescriptor or numeric identifier than can be used to match the returnedphysical block address. In another embodiment, the user submits severalinstances of write data 502 concurrently without a transactiondescriptor or numeric identifier and relies on the ordering or responsesto match returned physical block addresses.

In subsequent read operations 258, the storage users 500 refer to theindirection mechanism 200 to identify the particular SSD device 402 andphysical address in SSD array 400 where the read data 510 is located.Control element 300 reads the physical SSD device 402 referenced bydevice ID 232 at physical block address 234 and returns the read data510 to the particular one of the storage users 500.

The control element 300 checks block state 236 and might only performthe read operation if data has been written to the specified physicalblock 450. A block of some initial state (customarily all ‘0’s) would bereturned to the storage user 500 as the result of this invalid readoperation. In any embodiment wherein indirection mechanism 200 has noindirection entry 230, a similar block would be returned to the storageuser 500 indicating that no writes have occurred for the user addressthat maps to physical address of the specified physical block 450. Theaddress identified in indirection mechanism 200 is then used by thestorage users 500 to read data 510 from the SSD array 400.

Write Operation

Referring to FIGS. 1-4, the storage system 100 accepts write operationsof an integral number of blocks from storage users 500 but performswrites to the physical SSD array 400 in large blocks aggregated instaging buffers 370. The optimal size of the staging buffers 370 aredetermined experimentally and for the purpose of subsequent examples areassumed, through configuration, to be set to 4 MBs. For thisconfiguration, up to 1000 sub-blocks of 4 KBs can be contained withineach staging buffer 370. As explained above, performing large 4 MBwrites of uniform size from the storage system 100 to the SSD array 400improves the overall performance of the SSD array 400 since fewerdefragmentation operations are required later. As also explained above,a fewer number of larger block writes may increase write throughputcompared with a larger number of smaller random block writes.

Referring to FIGS. 1 and 4, to service write operations from any memberof storage users 500, storage system 100 uses control element 300 toidentify the most suitable indirect location for storing data andexecutes a sequence of operations to perform the write operation andupdate the indirection table 220.

The control element 300 maintains a device list 320 with informationregarding each physical SSD device 402 in SSD array 400. Each physicalSSD device 402 has a corresponding device buffer list 340 and acorresponding device block map 360. Control element 300 may consultdevice list 320 to determine the least utilized physical SSD device 402.

Utilization is considered in terms both of the number of physical blocks450 used in the SSD device 402 and the number of pending read operationsto the SSD devices 402. In one embodiment, the number of read operationsto specific 4 MB buffers 405 in the SSD devices 402 over some previoustime interval is also considered. This is explained below in FIGS.10-12. A high read utilization for a particular SSD device 402, such asSSD device 402A in FIG. 1, may cause the control element 300 to selectthe second SSD device 402B for a next block write, even when SSD device402A is currently storing less data. In some applications, there aresignificantly more reads from the SSD devices than writes into the SSDdevices. Therefore, evenly distributing read operations may require someSSD devices 402 to store significantly more data than other SSD devices.

Still referring to FIG. 4, after determining the optimal SSD device 402for writing, control element 300 consults device buffer list 340associated with the selected SSD device 402. The device buffer list 340contains a list of buffer entries 342 that identify free 4 MB buffers405 of storage in SSD array 400. Each buffer entry 342 represents thesame buffer size and contains separate block entries 345 that identifythe 4 KB blocks 450 within each F MB buffer 405 (FIG. 1). In oneembodiment, device buffer list 340 is maintained as a separate structurereferenced by the device entries in device list 320.

Device buffer list 340 has sufficient entries 345 to cover thecontiguous block space for each device entry 342 in device list 320.Each buffer entry 342 in device buffer list 340 contains minimally ablock map pointer 355 that points to a subset of bits 365 in the deviceblock map 360. In another embodiment, the buffer entries 342 may eachcontain a subset of the bits 365 from the device block map 360 thatcorrespond with a same 4 MB block in the same SSD device 402.

Device block map 360 contains a one to one mapping of 4 KB blocks 450(FIG. 1) for each buffer entry 342 in device buffer list 340. In thisexample, for a buffer entry 342 for a 4 MB 405 with 4 KB sub-blocks 450,each device block map 360 contains 1000 bits 365. Each bit 365represents the valid/invalid state of one 4 KB physical block 450 withina 4 MB physical buffer 450 in SSD array 400. Using the combination ofbuffer entry 342 and device block map 360, all unused or invalid 4 KBblocks 450 within the selected SSD device 402 for all 4 MB buffers 405in the SSD array 400 are identified.

Referring to FIG. 5, write operations 600 are submitted to the storagesystem 100 from one or more of the storage users 500. Staging buffer 370is selected as the next available buffer for the least utilized physicaldevice. Data for write operations A, B and Care copied into stagingbuffer 370 which is subsequently written to the SSD array 400 (FIG. 1).The write operations A, B, and C each include data and an associateduser address (write address). Other write operations may have occurredafter write operation C but before the write by control element 300 to aphysical disk in SDD array 400. When the 4 MB write to SSD array 400 iscompleted, indirection mechanism 200 is updated such that the logical 4KB blocks A, B and C point to valid indirection entries 230A, 230B and230C, respectively. These indirection entries maintain the mappingbetween the user address and the physical block address location 234 inthe SSD array 400 where the data A, B, and C is written.

In one embodiment, the block address 234 within each indirection entry230 is the exact physical address for the written blocks. In anotherembodiment, physical block addresses 234 are logical addresses derivedfrom the physical address. In another embodiment, block addresses 234are encoded with the device ID 232 (FIG. 1).

The control element 300 in FIG. 4 does not directly perform writes tothe selected SSD devices 402. A copy of the write data is placed in thestaging buffer 370 using as much space as necessary. Staging buffer 370is the same size as the 4 MB buffer entries 405 in the SSD array 400.Thus up to 1000 4 KB block writes can fit inside the staging buffer 370.Each 4 KB write from user 500 causes the corresponding bit 365 in deviceblock map 360 to be set. Multiple bits 365 are set for writes largerthan 4 KB.

Staging buffer 370 is written to the physical SSD device 402 in SSDarray 400 when the staging buffer 370 is full, nearly full, or apredetermined time has lapsed from the first copy into staging buffer370. Upon success of the write of the contents of the staging buffer 370into SSD array 400, the corresponding indirection entry 230 is updatedwith the physical address location (block address 234) of the data inSSD array 400. The indirection entry 230 is used in subsequent readoperations to retrieve the stored data.

To account for race conditions, an acknowledgement of the original writeoperation is not returned to the user 500 until the physical write intoSSD array 400 has occurred and the indirection mechanism 200 has beenupdated.

In one embodiment, the write data A, B, & C is copied into the stagingbuffer 370 by control element 300. In another embodiment, staging buffer370 uses references to the original write operation to avoid the need tocopy. In this case, staging buffer 370 maintains the list of links to beused by the write operation to SSD array 400.

Invalidation Operation

Through external factors, storage system 100 may periodically invalidatestorage or specific blocks of storage. This invalidation may be spawnedby activity such as deletion of data or expiration of cached informationinitiated by the storage users 500. In one embodiment, the granularityof the invalidation is the same as the granularity of the storage interms of block size. That is, invalidation occurs in integral number ofblocks (each 4 KB from the previous examples).

Invalidation clears the corresponding valid bit 365 in the device blockmap 360. For a specific storage block 450, device list 320 is consultedfor the appropriate device buffer list 340. The physical block address234 in indirection entry 230 is then used to determine the exact bit 365in the device block map 360 to clear. Once cleared, the indirectionentry 230 is updated to indicate that the entry is no longer valid.

The process of invalidation leaves unused 4 KB gaps within the 4 MBbuffers 450 of the SSD devices 402 which constitute wasted space unlessreclaimed. However, the entire 4 MB buffer 405 cannot be reclaimed aslong as other valid 4K blocks 450 are still stored within that 4 MBbuffer 405.

Remapping

To reclaim space freed during invalidation operations without losingexisting valid 4 KB blocks 450, control element 300 (FIG. 4)periodically reads all device buffer list entries 342 to determine ifmultiple 4 MB buffers can be combined. In one embodiment, suitabilityfor combination is determined through a count of the number of validblock entries 345 within each buffer entry 342. Each block entry 345 ina buffer entry 342 corresponds to a 4 KB block 450 within the same 4 MBbuffer 405 (FIG. 1). Combining more data from different buffers 405 intothe same buffer 405, increases the efficiency and capacity of read andwrite operations to the SSD array 400.

In a remapping operation, two or more 4 MB buffers 405 are read from theSSD array 400 and the valid 4 KB physical blocks 450 are copied into thesame empty 4 MB staging buffer 370. The 4 KB blocks 450 are packedsequentially (repositioned within the 4 MB staging buffer 370) such thatany holes created by the invalidated entries are eliminated. When all ofthe data from one or more 4 MB buffers 405 in SSD array 400 has beenread and processed into the same staging buffer 370, the staging buffer370 is written back into a same new 4 MB buffer 405 on the most suitableSSD device 402, determined again by referring to the device list 320.Upon completion of the write, the associated indirection entries 230 areupdated to reflect the new physical address locations for all of therepositioned 4 KB blocks 450. Upon completion of the update, all of theoriginally read 4 MB buffers 405 can be reused and are made available onthe corresponding device buffer list 340.

Remap Control and Optimization

One particular feature of the remapping operation is that a handshakingoperation is performed between the storage users 500 and the storagesystem 100. In one embodiment, the control element 300 of FIG. 4 sends aremap notification message to the storage users 500 prior to remappingmultiple different 4 KB blocks 450 from different 4 MB buffers 405 intothe same 4 MB buffer 405.

The remap notification message identifies the valid buffer entries 345that are being moved to a new 4 MB buffer 405. The physical data blocks450 that are being moved are committed in the new 4 MB buffer 405 in theSSD device 402 prior to the control element 300 sending out the remapnotification message to the storage users 500. The storage users 500then have to acknowledge the remap notification message before thecontrol element 300 can reclaim the 4 MB buffers 405 previously storingthe remapped 4 KB data blocks 450.

The storage users 500 acknowledge the remap notification message andthen update the indirection entries 230 in indirection mechanism 200 tocontain the new device ID 232 and new block addresses 234 for theremapped data blocks 450 (FIG. 1).

Defragmentation in prior SSD devices is typically done autonomouslywithout providing any notification to the storage users. The remappingdescribed above is transparent to the storage users 500 through thehandshaking operation described above. This handshaking allows thestorage users 500 to complete operations on particular 4 KB blocks 450before enabling remapping of the blocks into another 4 MB buffer 405.

In one optimization, the staging buffers 370 in FIG. 4 might only bepartially filled when ready to be written into a particular 4 MB buffer405 in SSD array 400. The control element 300 may take this opportunityto remap blocks 450 from other partially filled 4 MB buffers 405 in SSDarray 400 into the same 4 MB buffer where the current contents instaging buffer 370 are going to be written.

Similarly as described above, the control element 300 identifies free 4KB blocks in the new 4 MB buffer 405 via the device buffer list 340. Aremap notification message is sent to the storage users 500 for the datablocks 450 that will be copied into the staging buffer 370 and remapped.After the storage users 500 reply with an acknowledgement, all of thecontents of the staging buffer 370, including the new data and theremapped data from storage array 400, is written into the same 4 MBbuffer 405. This remaps the 4 KB blocks 450 from other sparse 4 MBbuffers 405 into the new 4 MB buffer 405 along with any new write datapreviously contained in the staging buffer 370.

In another optimization, there may not be many write operations 600currently being performed by the storage users 500. The control element300 may start reading 4 KB blocks 450 from SSD array 400 for one or moresparsely filled 4 MB buffers 405 into the staging buffer 370. Whenwrites 600 are received, the write data is loaded into the remainingfree blocks in the staging buffer 370. All of the contents in thestaging buffer 370 are then written into the same 4 MB buffer 405 afterthe remap acknowledge is received from the storage users 500. The blockspreviously read from the sparsely filled 4 MB blocks in the SSD arrayare then freed for other block write operations.

FIGS. 6-12 describe in more detail examples of how the storage system100 is used to remap and optimize storage usage in the SSD array 400. Asdescribed above, the SSD array 400 is virtualized into 4 MB buffers 405with 4 KB physical blocks 450. Thus, in this example, there will be 10244 KB physical blocks in each 4 MB buffer 405 in the SSD array 400. Ofcourse, other delineations could be used for the buffer size and blocksize within the buffers.

Referring to FIG. 6, the control element 300 in the storage system 100maintains a buffer entry 342 for each 4 KB data block 450 in each 4 MBbuffer 405 in SSD 400. The buffer entry 342 contains the pointer 355 tothe physical location of the 4 MB buffer 405 in SSD array 400. Differentcombinations of the 4 KB blocks 450 within the 4 MB buffer 405 mayeither contain valid data designated as used space or may contain emptyor invalid data designated as free space.

The control element 300 uses a register counter 356 to track of thenumber of blocks 450 that are used for each 4 MB buffer 405 and uses aregister counter 357 to track the number of times the blocks 450 areread from the same 4 MB buffer 405. For example, whenever a data iswritten into a previously empty buffer 405, the control element 300 willreset the value in used block count register 356 to 1024. The controlelement 300 will then decrement the value in used block count register356 for each 4 KB block 450 that is subsequently invalidated. Wheneverthere is a read operation to any 4 KB block 450 in a 4 MB buffer 405,the control element 300 will increment the value in a block read countregister 357 associated with that particular buffer 405.

The count value in register 357 may be based on a particular timewindow. For example, the number of reads in register 357 may be arunning average for the last minute, hour, day, etc. If the time windowwhere say 1 day, then the number of reads for a last hour may beaveraged in with other read counts for the previous 23 hours. If abuffer 405 has not existed for 24 hours, then an average over the timeperiod that the buffer has retained data may be extrapolated to anaverage per hour. Any other counting scheme that indicates the relativeread activity of a particular buffer 405 with respect to the otherbuffers in the SSD array 400 can also be used.

The device block map 360 as described above is a bit map where each bitindicates whether or not an associated 4 KB data block 450 in aparticular 4 MB buffer 405 is used or free. In the example, in FIG. 6, afirst group of bits 365A in the bit map 360 indicate that acorresponding first group of 4 KB blocks 450A in 4 MB buffer 405 areused. A second group of bits 365B in the bit map 360 indicate that acorresponding second group of 4 KB blocks 450B in buffer 405 are allfree, etc. Again, this is just one example, and the bits 365 can beconfigured to represent smaller or larger block sizes.

The overall storage system 100 (FIG. 1) performs three basic read,write, and invalidate data activities in SSD array 400. FIG. 7 shows inmore detail the write operations performed by the control element 300.In operation 600, the storage system 100 receives a user writeoperation. The control element 300 determines if there is a stagingbuffer 370 currently in use in operation 602. If not, the controlelement 300 initializes a new staging buffer 370 in operation 614 andinitializes a new buffer entry 342 for the data associated with thewrite operation in operation 616.

The control element 300 copies the user data contained in the writeoperation from the user 500 into the staging buffer 370 in operation604. The bits 365 in the device block map 360 associated with the dataare then set in operation 606. For example, the bits 365 correspondingto the locations of each 4 KB block of data in the 4 MB staging buffer370 used for storing the data from the user write operation will be setin operation 606. Operation 606 will also increment the used blockcounter 356 in buffer entry 342 for each 4 KB block 450 of data used inthe staging buffer 370 for storing user write data.

If the staging buffer 370 is full in operation 608, the control element300 writes the data in the staging buffer 370 into an unused 4 MB buffer405 in the SSD array 400 in operation 618. The control element 300 mayalso keep track how long the staging buffer 370 has been holding data.If data has been sitting in staging buffer 370 beyond some configuredtime period in operation 610, the control element 300 may also write thedata into the 4 MB buffer 405 in operation 618. The control element 300updates the indirection table 220 in FIG. 1 to include the SSD device ID232, user addresses 233, and block addresses 234 for the indirectionentries 230 associated with the data blocks 450 written into SSD array400. The process then returns to operation 600 for processing otherwrite operations.

FIG. 8 explains the operations performed by the control element 300 forread operations. In operation 630, the storage system 100 receives aread request from one of the users 500. The control device determines ifthe user read address in the read request is contained in theindirection table 220. If not, a read error message is sent back to theuser in operation 634.

When the read address is located, the control element 300 identifies thecorresponding device ID 232 and physical block address 234 (FIG. 1) inoperation 632. Note that the physical block address 234 may actuallyhave an additional layer of abstraction used internally by theindividual SSD devices 402. The control element 300 in operation 636reads the 4 KB data block 450 from SSD array 400 that corresponds withthe mapped block address 234. The read count value in register 357 (FIG.6) is then incremented and the control device returns to processingother read requests from the users 500.

FIG. 9 shows the operations that are performed by the control element300 for invalidate operations. The storage system 100 receives aninvalidate command from one of the users 500 in operation 642. Thecontrol element 300 in operation 644 determines if the user address 233in the invalidate request is contained in the indirection table 220(FIG. 1). If not, an invalidate error message is sent back to the userin operation 648.

When the address is successfully located in the indirection table, thecontrol element 300 identifies the corresponding device ID 232 andphysical block address 234 (FIG. 1) in operation 644. The controlelement 300 in operation 646 clears the bits 365 in the device block map360 (FIG. 6) that correspond with the identified block addresses 234.The used block counter value in register 357 is then decremented oncefor each invalidated 4 KB block 450. In operation 650, the controlelement 300 checks to see if the used block counter value in register356 is zero. If so, the 4 MB buffer 405 no longer contains any validdata and can be reused in operation 652. When the used block counter 356is not zero, the control element 300 returns and processes other memoryaccess requests.

FIGS. 10 and 11 show how data from different 4 MB buffers 405 in the SSDarray 400 are combined together. Referring first to FIG. 10, threedifferent buffer entries 342A, 342B, and 342C are identified by thecontrol element 300 for resource recovery and optimization. A rankingscheme identifies the best candidate buffers 405 for recover based onthe associated used block count value in buffer 356, the read countvalue in register 357 in the buffer entries 342 and a bufferutilization. One embodiment of the ranking scheme is described in moredetail below in FIG. 12.

In this example, the buffer entry 342A associated with 4 MB buffer 405Ahas an associated block count of 16 and a read count of 1. This meansthat the valid data A1 and A2 in buffer 405A has a combination of 16valid 4 KB blocks and has been read once. Sixteen different bits are setin the device block map 360A that correspond to the sixteen 4 KB validblocks of data A1 and A2.

The buffer entry 342B associated with 4 MB buffer 405B has a block countof 20 and a read count of 0, and the buffer entry 342C associated with 4MB buffer 405C has an associated block count of 24 and a read count of10. Similarly, 20 bits will be set in the device block map 360B thatcorrespond to the locations of the twenty 4 KB blocks of data B1 inbuffer 405B, and 24 bits will be set in the device block map 360C thatcorrespond to the twenty four 4 KB blocks of data C1 in buffer 405C.

The control element 300 combines the data A1 and A2 from buffer 405A,the data B1 from buffer 405B, and the data C1 from buffer 405C into afree 4 MB buffer 405D. In this example, the data A1 and A2 from buffer405A are first copied into the first two contiguous address ranges D1and D2 of buffer 405D, respectively. The data B1 from buffer 405B iscopied into a next contiguous address range D3 in buffer 405D after dataA2. The data C1 from buffer 405C is copied into a fourth contiguousaddress range D4 in buffer 405D immediately following data C1.

A new buffer entry 342D is created for 4 MB buffer 405D and the blockcount 356D is set to the total number of 4 KB blocks 450 that werecopied into buffer 405D. In this example, 60 total blocks 450 werecopied into buffer 405D and the used block count value in register 356Dis set to 60. The read count 357D is also set to the total number ofprevious reads of buffers 342A, 342B, and 342C. The device block map360D for buffer 405D is updated by setting the bits corresponding withthe physical address locations for each of the 60 4 KB blocks 450 ofdata A1, A2, B1 and C1 copied into buffer 405B. In this example, thedata A1, A2, B1 and C1 substantially fills the 4 MB buffer 405D. Anyremaining 4 KB blocks 450 in buffer 405D remain as free space and thecorresponding bits in device block map 360D remain set at zero.

The different free spaces shown in FIG. 10 may have previously containedvalid data that was then later invalidated. The writes to SSD array 400are in 4 MB blocks. Therefore, this free space remains unused until thecontrol element 300 aggregates the data A1, A2, B1, and C1 into anotherbuffer 405D. After the aggregation, 4 MBs of data can again be writteninto 4 MB buffers 405A, 405B, and 405C and the free space reused. Byperforming contiguous 4 MB writes to SSD array 400, the storage system100 reduces the overall write times over random write operations. Bythen aggregating partially used 4 MB buffers 405, the control element300 improves the overall utilization of the DDS array 400.

Referring to FIG. 11, the control element 300 ranks the 4 MB buffers 405according to their usefulness in operation 670. Usefulness refers to howmuch usage the storage system 100 is getting out of the data in the 4 MBbuffer 405. Again, ranking buffers will be explained in more detailbelow in FIG. 12. After the buffers are ranked, one of the stagingbuffers 370 (FIG. 4) is cleared for copying data from other currentlyused 4 MB buffers 405. For example in FIG. 10, a staging buffer 370 iscleared for loading data that will eventually be loaded into 4 MB buffer405D.

In operation 684, the control element 300 reads the information from thebuffer entry 342 associated with the highest ranked 4 MB buffer 405. Forexample, the information in buffer entry 342A and device block map 360Ain FIG. 10 is read. The control element 300 identifies the valid data inbuffer 405A using the associated buffer entry 342A and device block map360A in operation 686. The valid 4 KB blocks in buffer 405A are thencopied into the staging buffer 370 in operation 688. This process isrepeated in order of the highest ranked 4 MB buffers until the stagingbuffer (FIG. 5) is full in operation 674.

The control element 300 then creates a new buffer entry 342 in operation676 and sets the used block counter value in the associated register 356to the total number of 4 KB blocks copied into the staging buffer 370.For example, the control element 300 creates a new buffer entry 342D forthe 4 MB buffer 342D in FIG. 10. The control element 300 also sets thebits for the associated device block map 360D for all of the valid 4 KBblocks 450 in the new 4 MB buffer 405D.

In operation 678, the data in the staging buffer 370 is written into oneof the 4 MB buffers 405 in the SSD array 400 that is not currently beingused. For example, as described in FIG. 10, the aggregated data for A1,A2, B1 and B2 are stored in 4 MB buffer 405D of the SSD array 400. Thecontrol element 300 in operation 680 updates the indirection mechanism200 in FIG. 1 to include a new indirection entry 230 (FIG. 1) thatcontains the device ID 232 under user addresses 233 and correspondingphysical block addresses 234 for each of the 4K blocks in 4 MB buffer405D. The process then returns in operation 682.

Ranking Buffers

Because the SSD array 400 is used to tier data that is also stored inthe disk array 20 (FIG. 1), data in any of the 4 MB buffers 405 can bedeleted or “ejected” whenever that data has little usefulness beingstored in the SSD array 400. For example, storing data in the SSD array400 that is seldom read may have little impact in improving the overallread access time provided by the storage system 100 and is thereforeless useful. However, storing data in the SSD array 400 that isfrequently read could have a substantial impact in reducing the overallread access time provided by storage system 100 and is therefore moreuseful. Accordingly, the control element 300 may remove data from SSDarray 400 that is seldom read and replace it with data that is morefrequently read. This is different from conventional SSD devices thatcannot eject any data that is currently being used, regardless of theusefulness of the data.

FIG. 12 explains a scheme for determining what 4 MB buffers 405 torecover, and the criteria used for determining which buffers to recoverfirst. As explained above, a buffer 405 refers to a 4 MB section ofmemory in the SSD array 400 and a block 450 refers to a 4 KB section ofmemory space within one of the 4 MB buffers 405. Of course, the 4 MBbuffer size and the 4 KB block size are just examples and other bufferand block sizes could be used.

In operation 700, the control element 300 calculates the number of usedbuffers 405 in the SSD array 400 by comparing the number of bufferentries 342 with the overall memory space provided by SSD array 400.Operation 702 calculates the total number of 4 KB blocks 450 currentlybeing used (valid) in the SSD array 400. This number can be determinedby summing all of the used block counter values in each of the registers356 for each of the buffer entries 342.

The control element 300 in operation 704 calculates a fragmentationvalue that measures how much of the SSD array 400 is actually beingused. Fragmentation can be calculated globally for all buffer entries342 or can be calculated for a single 4 MB buffer 405. For example, thenumber of used blocks 450 identified in operation 702 can be divided bythe total number of available 4 KB blocks 450 in the SSD array 400. Afragmentation value close to 1 is optimal, and a value below 50%indicates that at least 2:1 buffer recovery potential exists.

Operation 708 calculates a utilization value that is a measure of howsoon the SSD array 400 will likely run out of space. A utilization above50% indicates the SSD array is starting to run out of space and autilization above 90% indicates the SSD array 400 in the storage system100 will likely run out of space soon. The control element 300determines the utilization value by dividing the number of used 4 MBbuffers 405 identified in operation 700 by the total number of available4 MB buffers 405 in SSD array 400.

If the utilization of the 4 MB buffers is less than 50% in operation708, no buffer ranking is performed, no buffers are discarded, and noblocks from different buffers are aggregated together in operation 714.In other words, there is still plenty of space in the SSD array 400available for storing additional data and space is not likely to run outsoon.

If the utilization is greater than 50% in operation 708, there is apossibility that the SSD array 400 could run out of space sometimerelatively soon. The control element 300 will first determine if thefragmentation value is greater than 50% in operation 710. Afragmentation less than 50% indicates that there are a relatively largepercentage of 4 KB blocks 450 within the 4 MB buffers 405 that arecurrently free/invalid and defragmenting the buffers 405 based on theirused block count values in registers 356 will likely provide the mostefficient way to free up buffers 405 in the SSD array 400.

In operation 716, the control element 300 ranks all of the 4 MB buffers405 in ascending order according to their used block count values intheir associated registers 356. For example, the 4 MB buffer 405 withthe lowest block count value in associated register 356 is ranked thehighest. The control element 300 then performs the defragmentationoperations described above in FIGS. 10 and 11 for the highest rankedbuffers 405. The results of the defragmentation my cause the utilizationvalue in operation 708 to fall back down below 50%. If not, additionaldefragmentation may be performed.

If the fragmentation value in operation 710 is greater than 50% inoperation 710, then defragmenting buffers is less likely to free upsubstantial numbers of 4 MB buffers 405. In other words, a relativelylarge percentage of 4 KB blocks 450 within each of the 4 MB buffers 405are currently being used.

Operation 712 first determines if the utilization is above 90%. If theutilization value is below 90% in operation 712, then the number of 4 MBbuffers is running out, but not likely to immediately run out. In thiscondition, the control element 300 in operation 718 will discard thedata in 4 MB buffers 405 that have a read count of zero in theassociated registers 357. This represents data in the SSD array 400 thathave relatively little use since it has not been used in read operationsfor a particular period of time.

A utilization value in operation 712 above 90% represents a SSD array400 that is likely to run out of 4 MB buffers 405 relatively soon. Thecontrol element 300 in operation 720 ranks the 4 MB buffers 405 inascending order according to the read counts in their associated readcount registers 357. For example, any 4 MB buffers 405 with a zero readcount would be ranked highest and any 4 MB buffers 405 with a read countof 1 would be ranked next highest. The control element 300 than discardsthe data in the 4 MB buffers 405 according to the rankings (lowestnumber of reads) until the utilization value in operation 712 dropsbelow 90%.

Note that defragmentation as described above in FIGS. 10 and 11 isfavored since data is compacted instead of being lost. If utilization isbelow 90% the control element 300 can alternatively discard the buffersthat have never been read for recovery.

Conventional SSD drives perform defragmentation to improve read accesstime however the capacity of the SSD drives remain the same. Theoptimization scheme described above increases memory capacity andimproves memory utilization by determining first if data blocks fromfragmented buffers can be combined together. When blocks from differentbuffers cannot efficiently be combined together, data is discarded basedon read activity. When the fast storage media begins to run out ofspace, the data most useful for improving memory access times is kept inthe fast storage media while other less useful data is accessed fromslower more abundant disc storage media.

The system described above can use dedicated processor systems, microcontrollers, programmable logic devices, or microprocessors that performsome or all of the operations. Some of the operations described abovemay be implemented in software and other operations may be implementedin hardware.

For the sake of convenience, the operations are described as variousinterconnected functional blocks or distinct software modules. This isnot necessary, however, and there may be cases where these functionalblocks or modules are equivalently aggregated into a single logicdevice, program or operation with unclear boundaries. In any event, thefunctional blocks and software modules or features of the flexibleinterface can be implemented by themselves, or in combination with otheroperations in either hardware or software.

Having described and illustrated the principles of the invention in apreferred embodiment thereof, it should be apparent that the inventionmay be modified in arrangement and detail without departing from suchprinciples. Any modifications and variation coming within the spirit andscope of the present invention are also claimed.

1. An apparatus, comprising: a storage media having a plurality ofbuffer regions configured to store copies of data stored in a storagearray; a staging buffer configured to buffer the data for differentwrite operations; a processor configured to aggregate together the datain the staging buffer from the different write operations and store theaggregated data into blocks of a buffer region of the plurality ofbuffer regions of the storage media; wherein the processor is furtherconfigured to discard data in blocks of the buffer region having aminimum read count when a number of buffer regions currently being usedis below a first threshold number and discard data in the buffer regionsaccording the a ranking of buffer regions when the number of bufferregions currently being used is above a threshold.
 2. The apparatus ofclaim 1, an indirection table configured to map the aggregated data fromthe write operations to different block regions within a same one of thebuffer regions; wherein the write operations are each allocatedindirection entries within the indirection table and the indirectionentries include device identifiers for storage devices in the storagemedia and physical addresses of the block regions where the data isstored in the storage media.
 3. The apparatus of claim 1, wherein a sizeof the buffer regions and a size of the block regions are configurable.4. The apparatus of claim 3, wherein the processor is configured toselect the size of the buffer regions to increase write throughput tothe storage media.
 5. The apparatus of claim 1, wherein the processor isconfigured to aggregate different groups of data for different groups ofwrite operations into a staging buffer and write the differentaggregated groups of data from the staging buffer into associated bufferregions within the storage media.
 6. The apparatus of claim 1, whereinthe indirection table maps random addresses of the write operations intocontinuous block address locations within the buffer regions.
 7. Theapparatus of claim 1, wherein the storage media comprises an array ofSolid State Devices (SSDs).
 8. The apparatus of claim 1, wherein theprocessor is configured to discard data from the buffer regions andreplace the data discarded from the buffer regions with other data fromthe storage array.
 9. The apparatus of claim 8, further comprising blockcounters containing block count values identifying a number of the blockregions in the different buffer regions containing valid data, whereinthe processor is configured to discard data from the buffer regions oraggregate data from the different buffer regions together into a sameone of the buffer regions according to the block count values.
 10. Theapparatus according to claim 1, further comprising bit maps for each ofthe buffer regions, where bits in the bit maps identify a used or unusedstatus for data within associated block regions within buffer regions,and the processor is further configured to combine data from thedifferent buffer regions together into a same one of the buffer regionsaccording to the bit maps.
 11. An apparatus, comprising: a storagemedia; a processor configured to aggregate together data from differentwrite operations and store the aggregated data into one of multipledifferent buffer regions of the storage media; and an indirection tableconfigured to map the aggregated data from the write operations todifferent block regions within a same one of the buffer regions, and theprocessor is further configured to map the aggregated data from thewrite operations to different block regions within a same one of thebuffer regions and to track read count values associated with thedifferent buffer regions and to discard data from the buffer regionsaccording to the associated read count values.
 12. A storage system,comprising: a storage media; a control element configured to: identifybuffer regions within the storage media that store groups of data incontiguous address locations; and the storage media is configured to:operate as a tiering media for providing copies of the data contained ina storage array in response to read operations; remap the data from theblocks of different buffer regions within the storage media into a sameone of the buffer regions; discard the data from the buffer regionsaccording to utilization of the buffer regions; and replace the datadiscarded from the buffer regions in the storage media with other data.13. The storage system of claim 12, wherein the utilization of thebuffer regions corresponds with a number of the buffer regions that arecurrently being used in the storage media and a number of buffer regionsthat are available to be used in the storage media.
 14. The storagesystem of claim 12, further comprising block counters associated withthe different buffer regions, the block counters identifying a number ofused blocks in the associated buffer regions.